Content annotation tool

ABSTRACT

A content annotation tool is disclosed. In a configuration, a portion of a movie may be obtained from a database. Entities, such as an actor, background music, text, etc. may be automatically identified in the movie. A user, such as a content producer, may associate and/or provide supplemental content for an identified entity to the database. A selection of one or more automatically identified entities may be received. A database entry may be generated that links the identified entity with the supplemental content. The selected automatically identified one or more entities and/or supplemental content associated therewith may be presented to an end user.

BACKGROUND

Modern movie or video browsing may allow a user to interact with contentdisplayed. For example, a DVD may have selectable icons that aredisplayed and, when selected, the icons may cue display of directorcommentary, for example. Online video sharing websites may allow usersto insert comments that may be tagged to a particular time reference ofthe video. Video content creators may also insert URLs into videos thatmay link to content external to the website hosting the video. However,the tools to generate interactive content are limited. For example, insome instances, a content creator may be required manually indicate anactive or clickable portion of the screen. The active portion of thescreen may not coincide with the object of interest spatially and/ortemporally.

BRIEF SUMMARY

According to an implementation of the disclosed subject matter, aportion of a first movie may be obtained. An entity within the portionof the movie may be automatically identified. Supplemental content maybe received from a first user about the identified entity within thefirst movie. A database entry may be stored that links the identifiedentity within the portion of the first movie and the supplementalcontent about the identified entity. A portion of a second movie may beprovided to a second user. The portion of the second movie may includethe identified entity. A request may be received from the second userbased upon the portion of the second movie and the identified entity.Responsive to the request, the supplemental content may be provided tothe second user.

In an implementation, a movie may be provided. A query from a user maybe received. The query may be received by an input method duringplayback of the movie. A location reference for the query may bedetermined. The location reference may identify an entity in the movie.Supplemental content about the identified entity may be retrieved from adatabase. The database may include one or more automatically identifiedentities for one or more movies and supplemental content. A response tothe query may be provided to the user. The response may includesupplemental content.

In an implementation, a query may be received by a database. Thedatabase may include one or more automatically identified entities fromone or more movies and supplemental content. A response may be generatedthat includes supplemental content and is based on at least one of auser history and a user location. The response may be sent to a device.

In an implementation, supplemental content may be received. Thesupplemental content may be stored. Content may be obtained. An entityin the content may be automatically identified. A portion of thesupplemental content may be retrieved based on the entity. A databaseentry that links the identified entity with a portion of thesupplemental content while the entity is present in the content may bestored. A portion of the content that includes the identified entity maybe provided to a user. A request may be received from the user basedupon the portion of the movie and the identified entity. Responsive tothe request, the supplemental content may be provided to the user.

In an implementation, a portion of a first movie may be obtained. One ormore entities within the portion of the movie may be automaticallyidentified using a technique such as face recognition, audiorecognition, voice recognition, text recognition, or the like.Supplemental content may be received from a first user about one or moreof the automatically identified entities within the first portion of themovie. The supplemental content may include a selection of one or moreof the automatically identified entities, and/or additional contentprovided by a user. A database entry may be stored and/or generated thatlinks the identified entity within the portion of the first movie andthe supplemental content about the identified entity. Supplementalcontent may be received and/or associated with the selected one or moreentities. In some instances, the supplemental content and/orautomatically identified entities may be associated with a timereference.

Additional features, advantages, and implementations of the disclosedsubject matter may be set forth or apparent from consideration of thefollowing detailed description, drawings, and claims. Moreover, it is tobe understood that both the foregoing summary and the following detaileddescription provide examples of implementations and are intended toprovide further explanation without limiting the scope of the claims.Implementations disclosed herein may allow content creators to submitsupplemental content that may be provided to an end user. Thesupplemental content may be interactive with a user's input actions(e.g., pausing a movie, selecting content in a video frame, etc.), andmay be updated and/or provided to more than one video. These and otherfeatures may allow for generation of interactive media and content basedupon entities present within the media.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are included to provide a furtherunderstanding of the disclosed subject matter, are incorporated in andconstitute a part of this specification. The drawings also illustrateimplementations of the disclosed subject matter and together with thedetailed description serve to explain the principles of implementationsof the disclosed subject matter. No attempt is made to show structuraldetails in more detail than may be necessary for a fundamentalunderstanding of the disclosed subject matter and various ways in whichit may be practiced.

FIG. 1 shows a computer according to an implementation of the disclosedsubject matter.

FIG. 2 shows a network configuration according to an implementation ofthe disclosed subject matter.

FIG. 3 shows an example process of associating an entity withsupplemental content according to an implementation disclosed herein.

FIG. 4 shows an example process of providing supplemental content withan identified entity according to an implementation disclosed herein.

FIG. 5 is an example process of generating a response to a query asdisclosed herein.

FIG. 6 shows an example arrangement of devices according to animplementation disclosed herein.

DETAILED DESCRIPTION

According to the present disclosure, a tool is provided that allowsanyone to create an interactive video, audio, or multimedia sequencethat may extend beyond consumption of the content itself. For example,the tool may be used to identify actors in a movie using facialrecognition and link or attach supplemental information about each actoridentified. For example, the supplemental information may be containedin a database, and the tool may create a link between one or moreactors' faces and the location of a subset or all of the supplementalinformation in the database corresponding to each of the actors. Whileviewing a video, a user may select a face of one of the actors byclicking on it, for example. The actor's biography may be presented tothe user. A URL to other locations that may contain more informationabout the actor may be provided as well. As another example, a movieproducer may have interesting facts or anecdotes about a scene,alternate scenes or outtakes, or advertising links or content. Thissupplemental content may also be inserted and made responsive to anappropriate cue (e.g., selection of an actor). Similarly, a consumer mayutilize an implementation disclosed herein to embed URLs or supplementalcontent in their home videos such as identifying family members, pets,etc.

An implementation may provide a do-it-yourself tool for an originalcontent maker to supply information and hyperlink it within the content.The content producer may embed supplemental content such as Internetlinks that could be triggered on a user's mobile device based on theuser's location. A user's device may receive a notice that the user isproximal to a location where a scene was filmed for a movie the userrecently watched. For example, a content maker may input that movie ABCwas filmed in Paris, France. The information may be stored to adatabase. If a user's mobile device is in proximity to Paris or a morespecific location in Paris where ABC was filmed, the user may receive anotice on the mobile device indicating that ABC was filmed nearby.

As another example, the filming locations for a movie may be displayedon a map that the user can interact with using navigation, for example.As another example, a user may pause a video, and select a particularpiece of clothing or accessory worn by an actor in the scene or that ispresent as a prop (e.g., a soda can). A selection may be made by, forexample, drawing a box around the area, person, or item of interest or amouse click. The user's selection of an area, person, or item (e.g., anactor, clothing, accessory, prop, etc.) may cause a hyperlink to appearthat may link the user to a webpage where the clothing, item, and/oraccessory is identified and/or can be purchased. In some configurations,pictures and/or other content involving the selected entity may bepresented to the user or hyperlinked. Thus, the annotations of thecontent may be stored to a database that may be queried in response to,for example, a user specific query or a user location. The consumptionof the content may not be concurrent with the presentation of suchsupplemental content. For example, at a first time, a user may view amovie that includes supplemental content provided or selected by acontent producer. At a second time, distinct from the first time, theuser may be presented with an advertisement for an article of clothingthat was in the movie the user viewed. The presentation of the articleof clothing may be based on the user having viewed the movie with anactor wearing the specific clothing item and the user being in alocation proximal to a store that offers the clothing item.

Implementations of the presently disclosed subject matter may beimplemented in and used with a variety of component and networkarchitectures. FIG. 1 is an example computer 20 suitable forimplementations of the presently disclosed subject matter. The computer20 includes a bus 21 which interconnects major components of thecomputer 20, such as a central processor 24, a memory 27 (typically RAM,but which may also include ROM, flash RAM, or the like), an input/outputcontroller 28, a user display 22, such as a display screen via a displayadapter, a user input interface 26, which may include one or morecontrollers and associated user input devices such as a keyboard, mouse,and the like, and may be closely coupled to the I/O controller 28, fixedstorage 23, such as a hard drive, flash storage, Fibre Channel network,SAN device, SCSI device, and the like, and a removable media component25 operative to control and receive an optical disk, flash drive, andthe like.

The bus 21 allows data communication between the central processor 24and the memory 27, which may include read-only memory (ROM) or flashmemory (neither shown), and random access memory (RAM) (not shown), aspreviously noted. The RAM is generally the main memory into which theoperating system and application programs are loaded. The ROM or flashmemory can contain, among other code, the Basic Input-Output system(BIOS) which controls basic hardware operation such as the interactionwith peripheral components. Applications resident with the computer 20are generally stored on and accessed via a computer readable medium,such as a hard disk drive (e.g., fixed storage 23), an optical drive,floppy disk, or other storage medium 25.

The fixed storage 23 may be integral with the computer 20 or may beseparate and accessed through other interfaces. A network interface 29may provide a direct connection to a remote server via a telephone link,to the Internet via an internet service provider (ISP), or a directconnection to a remote server via a direct network link to the Internetvia a POP (point of presence) or other technique. The network interface29 may provide such connection using wireless techniques, includingdigital cellular telephone connection, Cellular Digital Packet Data(CDPD) connection, digital satellite data connection or the like. Forexample, the network interface 29 may allow the computer to communicatewith other computers via one or more local, wide-area, or othernetworks, as shown in FIG. 2.

Many other devices or components (not shown) may be connected in asimilar manner (e.g., document scanners, digital cameras and so on).Conversely, all of the components shown in FIG. 1 need not be present topractice the present disclosure. The components can be interconnected indifferent ways from that shown. The operation of a computer such as thatshown in FIG. 1 is readily known in the art and is not discussed indetail in this application. Code to implement the present disclosure canbe stored in computer-readable storage media such as one or more of thememory 27, fixed storage 23, removable media 25, or on a remote storagelocation.

FIG. 2 shows an example network arrangement according to animplementation of the disclosed subject matter. One or more clients 10,11, such as local computers, smart phones, tablet computing devices, andthe like may connect to other devices via one or more networks 7. Thenetwork may be a local network, wide-area network, the Internet, or anyother suitable communication network or networks, and may be implementedon any suitable platform including wired and/or wireless networks. Theclients may communicate with one or more servers 13 and/or databases 15.The devices may be directly accessible by the clients 10, 11, or one ormore other devices may provide intermediary access such as where aserver 13 provides access to resources stored in a database 15. Theclients 10, 11 also may access remote platforms 17 or services providedby remote platforms 17 such as cloud computing arrangements andservices. The remote platform 17 may include one or more servers 13and/or databases 15.

More generally, various implementations of the presently disclosedsubject matter may include or be implemented in the form ofcomputer-implemented processes and apparatuses for practicing thoseprocesses. Implementations also may be implemented in the form of acomputer program product having computer program code containinginstructions implemented in non-transitory and/or tangible media, suchas floppy diskettes, CD-ROMs, hard drives, USB (universal serial bus)drives, or any other machine readable storage medium, wherein, when thecomputer program code is loaded into and executed by a computer, thecomputer becomes an apparatus for practicing implementations of thedisclosed subject matter. Implementations also may be implemented in theform of computer program code, for example, whether stored in a storagemedium, loaded into and/or executed by a computer, or transmitted oversome transmission medium, such as over electrical wiring or cabling,through fiber optics, or via electromagnetic radiation, wherein when thecomputer program code is loaded into and executed by a computer, thecomputer becomes an apparatus for practicing implementations of thedisclosed subject matter. When implemented on a general-purposemicroprocessor, the computer program code segments configure themicroprocessor to create specific logic circuits. In someconfigurations, a set of computer-readable instructions stored on acomputer-readable storage medium may be implemented by a general-purposeprocessor, which may transform the general-purpose processor or a devicecontaining the general-purpose processor into a special-purpose deviceconfigured to implement or carry out the instructions. Implementationsmay be implemented using hardware that may include a processor, such asa general purpose microprocessor and/or an Application SpecificIntegrated Circuit (ASIC) that implements all or part of the techniquesaccording to implementations of the disclosed subject matter in hardwareand/or firmware. The processor may be coupled to memory, such as RAM,ROM, flash memory, a hard disk or any other device capable of storingelectronic information. The memory may store instructions adapted to beexecuted by the processor to perform the techniques according toimplementations of the disclosed subject matter.

In an implementation, an example of which is provided in FIG. 3, aportion of a first movie may be obtained at 310. For example, anindividual may upload a home movie or a movie company may provide amovie to a database. In general, a content producer may make the movieavailable directly (e.g., uploading to the database) or indirectly(e.g., requiring connection to a separate database via the Internet). Acontent producer may be an individual who determines what, if any,supplemental content may be rendered in a movie. A movie may refer toany video content including, but not limited to, a portion of a video, auser-generated video, or other multimedia content, and it may includesupplemental content as disclosed herein.

An entity within the portion of the movie may be automaticallyidentified at 320. An entity may be an audio component of the movie, avisual component of the movie, or a combination thereof. Examples of anaudio component may include, without limitation: a song, a soundtrack, avoice or speech, and a sound effect. A sound effect may refer to a dogbarking, a car screech, an explosion, etc. A visual component mayinclude, for example: a scene break, a geographic location, a face, aperson, an object, a physical object, a text, or a landmark. Ageographic location may refer to a particular place such as Paris, anaddress, a landmark such as the Grand Canyon, etc. A face may bedetermined from a gallery in which a person has been tagged, identified,or otherwise labeled. For example, a home video application may identifyfaces of individuals in a video. In some instances, an individual may beidentified in an online photo or other type of online publication ornews article. Such sources may also be utilized to automaticallyidentify a visual component. An example of an object that may beautomatically identified is a car. The car may be identified by itsmake, model, manufacturer, year, etc. Faces, objects, and other entitiesmay be identified by comparison to related galleries or other storedimages that include those entities, such as where a face in a home videois identified based upon a gallery maintained by a user that includesimages of a person present in the home video. Similarly, a car may beidentified by comparison to a database of images of known makes andmodels of automobiles. A movie may contain text, for example, asubtitle, a closed caption, or on a sign in the movie. OCR may beemployed to identify the text that is available in a particular scene orframe of the movie.

Automatic identification of an entity in the movie may be performedusing, for example, facial recognition, speech or voice recognition,text recognition or optical character recognition, or patternrecognition such as for a song. Automatic identification of one or moreentities may proceed in real time as frames of a movie are received,played, or streamed. In some configurations, the identification may belimited to a particular entity such as an actor. In some configurations,a database may be used to store features associated with variousobjects, persons, faces, geographic locations, etc. to which a moviethat is obtained 310 may be compared to identify one or more entities.Multiple entities may be determined simultaneously in a movie. An entitymay be determined on a frame-by-frame basis or based on a chapter/scenebreak. Information regarding an automatically identified entity may bestored in a database and be linked to a time reference or otherreferences to the movie. The database containing the stored entity mayreceive queries regarding the automatically identified entity. Forexample, a song may be identified as being present from one timereference to another or in a particular chapter or scene of a movie.Thus, automatic identification of an entity may include, withoutlimitation, automatically recognizing: faces, objects (e.g., cars,airplanes, bikes, etc.), geographic locations, logos, gender, age,characteristics of an individual (e.g., whether a person is smiling orfrowning), animals, speech, printed text, scene breaks in a movie,object tracking, speech recognition, or a song/music (e.g., audiorecognition). Object tracking may refer to detecting an object over twoor more time reference points. For example, a soda can may be detectedin a first scene and the same soda can may be detected throughout thefirst scene and/or when the soda can again appears in a second scene,which may or may not chronologically follow the first scene.

Supplemental content may be received from a first user, such as acontent producer, about the identified entity within the first movie orselected from an automatically identified entity by the first user at330. Supplemental content may refer to, for example: a text, an audioentity, a visual entity, a URL, a picture, an advertisement, and alocation. For example, an actor may have been automatically identifiedas part of the process referred to earlier at 320. A content producer,director, or the like may select the identified actor and provide ananecdotal story that occurred during filming that involved the actor.Subsequently, the anecdote may appear when a user selects the actor,such as while the user is viewing the movie, or when the user performs aweb-based query related to the actor or the movie. The anecdote and/orthe identified actor may be supplemental content as disclosed herein. Asanother example, a picture may be provided that conveys abehind-the-scenes shot of a particular moment in a movie. When the movieis viewed by a user, the picture may be provided to the user at a timewhen the user is viewing the associated moment in the movie, and/or inresponse to a request from the user for supplemental content associatedwith that moment or with the movie generally.

The first user may elect to have a subset of all automaticallyidentified content displayed to a user upon a particular action from theuser, such as pausing the movie, or selecting an area or point of thescreen to indicate a desire to select an entity such as an actor, anarticle of clothing a prop, an item, or the like. A movie may containdifferent subsets of supplemental content that are presented to a userat different time reference points. For example, a content producer mayelect to have actors identified when a user pauses the movie at a firsttime reference that corresponds to a first scene. At a second timecorresponding to a second scene, but where the second scene containsentities that overlap with those in the first scene reference, thecontent producer may have only information about inanimate objectspresented at this time. An automatically identified entity may havedifferent supplemental content provided and/or associated with it atdifferent time references as well. For example, at a first timereference, the content producer may include, as supplemental content, ananecdote that appears if the user selects an actor. At a second timereference, the content producer may include other supplemental contentsuch as a hyperlink to the clothing and/or accessories worn by the actorpresented to the user, or the user may be directly linked to a webpagewhere such clothing and/or accessories can be viewed and/or purchased.More generally, the content producer may select some of theautomatically identified content as supplemental content as disclosedherein, and/or may provide additional information in association withautomatically identified content as supplemental content.

In some configurations, a time reference may be associated with thesupplemental content. For example, the anecdote described in an earlierexample may be associated with the particular actor or the duration ofthe scene to which it pertains. A database entry may be stored thatlinks the identified entity within the portion of the first movie andthe supplemental content about the identified entity at 340. Thedatabase entry may be queried independent of viewing of the movie.Continuing the example, if a person searches the name of the actor forwhom the director conveyed an anecdote, the anecdote may appear in thesearch results and be associated with the movie and/or a time reference.The database entry may be based upon an identification of the portion ofthe movie, the supplemental content, and the identified entity to adatabase.

For any of the implementations disclosed herein, entity identificationmay not be limited to a particular time reference. For example, facialrecognition may identify the face of an actor wearing a blue shirt in afirst scene as the same actor in a second scene in which the actor iswearing a gray coat. If a content maker inputs the actor's name, theactor will be so identified in both the first scene and the secondscene. Thus, information received for an identified entity may beassociated with the entity in any and/or all instances of the entitywithin the movie or video.

In some instances, an entity may not be identified. For example, theidentity of an “extra” in a big-screen movie production may not beknown. A content producer or user may indicate the identity of theentity. In some configurations, when a user identifies an entity, it maybe vetted by the content producer or by a group moderated process wheremultiple users confirm, reject, or suggest an alternative identity forthe identification provided. In some instances, supplemental content maybe provided independent of the movie itself. For example, the firstmovie may contain a scene involving a particular bar in Boston. A usermay visit the bar where the scene was shot and link a picture of theexterior/interior of the bar to the scene in the movie or the movieitself. Thus, over time, a database may accumulate information fromusers or content and the identified entities of one or more movies.

A portion of a second movie may be provided to a second user, whichincludes the identified entity and/or supplemental content at 350. Forexample, a second user may elect to watch a second movie that containsan actor who was identified in the first movie and/or had somesupplemental content associated with the actor or scene. In someinstances, the first movie and the second movie may be the same. Forexample, the second user may watch the first movie. Similarly, theanecdote and/or other supplemental content may be provided to a userthat views another movie or other media in which an entity is identifiedand for which supplemental content is received. For example, if theexample anecdote relates to an actor, the anecdote may be provided to auser who views a second, different movie in which the actor appears.

A request from the second user may be received based upon the portion ofthe second movie and the identified entity at 360. For example, a usermay select an object in the scene of the second movie by clicking on theobject, pausing the movie, drawing a box around an area of the movieeither while it is playing or while it is paused, inputting text such asan actor's name or a chapter/scene number, or speaking a name or othervoice command. A processor connected to the database that stored theentry may receive the query. In some instances, the database may belinked to a server that receives the query and subsequently queries thedatabase.

In some configurations, the entity may be tracked in the moviethroughout the duration of time that the entity exists within theportion of the movie. For example, an actor's or object's position in ascene may be communicated to a database as a series of coordinates alongwith information to indicate the actor's name or the object's identity,such as a soda can, and a time reference or time index. The actor orobject may be identified for a portion of the movie such as a scene orfor the entirety of the movie. In the example where coordinates arerelayed to convey the position or dimension of the actor or object in aportion of the movie, the receipt of a user query on a particular imageframe of a movie or during playback of the movie may be compared to thestored coordinates and time index or reference to identify the objectbeing queried.

In some instances, a user may draw a box to make a selection of peopleand/or objects. For example, a scene may involve four individuals, eachwith an object in hand. A user may draw a circle around each actor thatencompasses the object each actor possesses. In some configurations, thesystem may assume that the user intends to have it track the actors orobjects alone. In other instances, a window for each selected object mayappear and provide supplemental content that is available, if any, forthe object. In some configurations, the user may receive an indicationthat multiple actors, objects, etc. have been selected and the user mayselect the actors, objects, etc. that the user would like to havequeried or tracked or for which the user would like supplemental contentpresented. For example, the user may be presented with a selectable listof the actors, objects, etc.

Automatically identified entities may include audio or visualinformation that is not directly selectable by an input device such as amouse. For example, a user may be interested in a song that is beingplayed during a particular scene. The user may, for example, pause themovie and a sound icon may appear which, when clicked, may display thesong currently being played. As another example, the user may type theword “song” using a keyboard and the title of the song may appear. Theuser may click on the title of the song to reveal additionalsupplemental content such as a link to the movie soundtrack. Similarfeatures may be used with other entities that may not bedirectly-selectable, such as a chapter number, scene name, etc.

Responsive to the request from the second user, the supplemental contentmay be provided to the second user at 370. For example, a user may clickon an actor's face in the movie. The movie may pause and informationabout other movies the actor has been involved with, the latest newsregarding the actor, upcoming movies in which the actor will appear, theactor's personal webpage or comment feed, and/or other supplementalcontent may also be displayed. Supplemental content may be presented inthe form of a URL. For example, if the user selects the comment feed, itmay link the user to an application or webpage external to themovie-providing application or webpage.

An example of an implementation is provided in FIG. 4. A movie may beprovided at 410 by, for example, a streaming service, an application, ora stand-alone player. A query may be received from a user at 420. Thequery may be received by an input method during playback of the movie.An input method may refer to use of an input device such as a keyboardor a mouse, a touch-interaction, a gesture, or a voice command. Forexample, a user may speak an actor's name or a chapter/scene number. Alocation reference that identifies an entity for the query may bedetermined at 430. The location reference may identify an entity in themovie and it may refer to a particular location of the entity with aframe or sequence of frames of the movie (e.g., the area occupied by theentity, the time during which the entity appears, etc.). As describedearlier, the query may be associated with a time reference andcoordinates. The time reference and coordinates may be used to query adatabase for automatically identified entities and/or supplementalcontent. Thus, a user may obtain the identity of one or more selected orlocated automatically identified entities by speaking the query,selecting an area or point on a display, and/or making a gesture to anarea of the display. Gesture recognition, voice recognition, andconventional mechanisms of selecting an entity or querying a databasemay be utilized with any implementation disclosed herein.

In some configurations, the supplemental content about the identifiedentity may be retrieved from the database at 440. The database mayinclude one or more automatically identified entities for a one or moremovies and supplemental content. A response to the query that includessupplemental content may be provided to the user at 450. In someinstances, the supplemental content may be provided substantially nearthe location of the reference. For example, if the query is provided bya mouse click to an actor's face, the supplemental content regardingthat particular actor may be provided adjacent to the actors face. Thesystem may track the actor throughout the scene and during that time,the supplemental content may continue to be displayed adjacent to theactor's face.

An example of an implementation is provided in FIG. 5. A query may bereceived by a database at 510. The database may contain entriescorresponding to one or more automatically identified entities for oneor more movies and supplemental content. A response may be generatedthat includes supplemental content based on at least user history and/oruser location at 520. In some instances, a user history may includecontent consumed by a user (e.g., a song that was purchased, a moviethat was viewed) and in some instances, the user history may includecontent for which the user has indicated an interest (e.g., a book thatis in purchase queue). The response may be sent to a device such as amobile phone, a smartphone, a laptop, or any other computing device at530.

For example, a user history may indicate that a user viewed movie Yankeethat was filmed at restaurant Zulu. The user may submit a web query on amobile device for restaurants near the user's location. The query may bereceived by a processor connected to the database. The user's locationmay be used to identify and generate a list of restaurants near theuser, which is then cross-referenced against the user's history tofurther narrow the list. For example, Zulu may be suggested to the userand be presented with supplemental content indicating that Yankee, amovie the user has seen or may be interested in seeing, was filmed atZulu. Additional information may be obtained from the user or othersources that provides additional relevant information, such as a userpreference. For example, it may be determined based upon a user responseor other profile information available about the user, that the user maybe a vegetarian. Restaurants that lack a well-received vegetarian menuor option may be excluded from the response or from a list generated byquerying a database. In some instances, such a query may beautomatically sent by the device.

In some configurations, an augmented reality may be provided to thedevice. For example, a user visiting the Grand Canyon on a vacation mayreceive a notice that includes supplemental content related to the GrandCanyon. For example, based on a user history or a user preference, itmay be determined that the user is interested in a particular topic,and/or movie/music genre. Information may be provided to the userrelated to the user's interest. For example, the user may be a historyaficionado as determined by the user's voluminous collection ofhistorical documentaries. The user may receive a notice containinglesser-known interesting historical facts about the Grand Canyon, orabout a particular location or feature at the Grand Canyon that is closeto the user's location. In some configurations, the user may be able toaugment the reality viewed by viewing a location, such as the GrandCanyon, using an electronic device such as a camera connected to amobile phone. The Grand Canyon may be shown on the display of the devicewith one or more selectable markers. If the user selects a marker, theuser may be provided supplemental content. For example, if the South Rimis on the display and the user selects a marker on the South Rim, theuser may be provided facts about the Grand Canyon Railway. A display maybe augmented without the use of a selectable marker. For example, theuser may touch the South Rim on the display to have supplemental contentprovided. If markers are provided on the display, they may adapt as theuser moves or pans the device around an area. The user may also receivea recommendation or suggestion of content related to the Grand Canyon.For example, a documentary that the user has not seen, possibly onerelated to the Grand Canyon Railway, may be suggested to the user forpurchase or download. Thus, a user's location, a user's history, and/orsupplemental content may be used to augment the reality of the user.

In some configurations, an update to the supplemental content from thedevice may be received, for example, by a processor connected to adatabase. For example, a fire may have destroyed a building where moviewas filmed. A user may take a picture of the burned-out building andupload it to the particular movie scene containing the building. Thepicture may be tagged with other information such as a link to a newsstory about the fire, the date, the author of the picture, etc.Subsequent to the upload of the picture, other users who receive thesupplemental content for that particular movie may be presented with thepicture.

In an implementation, supplemental content may be received and it may bestored, for example, to a computer readable medium. Content such as amovie or song may be obtained. An entity in the content may beautomatically identified and/or tracked in the content throughout theduration that the entity exists within the content as disclosed herein.The tracking may be a dynamic process, for example, that determines thepresence of the entity and may relay coordinates of the entity on thedisplay or a time reference or a scene or chapter number. A portion ofthe supplemental content may be retrieved based on the automaticallyidentified entity as disclosed herein, such as from a database entrythat links the identified entity with a portion of the supplementalcontent while the entity is present in the content. A link, for example,may refer to a table containing at least the identified entity and theportion of the supplemental content. A portion of the content thatincludes the identified entity may be provided to a user as disclosedherein. A request from the user based upon the portion of the movie andthe identified entity may be received, in response to which thesupplemental content may be provided to the user as previouslydescribed.

A content annotation tool as disclosed herein may be provided, forexample, via a hosted, cloud-based, or other remote system, such as ahosted system that provides media content to a user via a web browser orsimilar interface. Alternatively or in addition, a tool as disclosedherein may be provided by way of a local computer that implements thefunctionality disclosed herein, which may communicate with a remotesystem to provide and/or obtain supplemental content as disclosed. Insome configurations, a remote system may provide multiple users withaccess to movies and other media, and may then aggregate varioussupplemental content provided by the multiple users as disclosed. Theannotation tool may be integrated with a database which storessupplemental content as described, or it may be provided by a separatecomputer system that is in communication with one or more databases thatstore and provide media and/or supplemental content. An example of theannotation tool is provided in FIG. 6. The annotation tool 610 may beconnected to one or more databases 15 via a network 7, which may storethe various media, supplemental content, and other data disclosedherein. In some configurations, the annotation tool 610 may bevirtualized and provided to clients 10, 11 via the network 7. Clients10, 11 may provide content stored on the client 10, 11 computing deviceand/or the database 15 as disclosed herein.

In an implementation, a portion of a first movie may be obtained. Asdescribed earlier, a movie may be any video content provided from anysource (e.g., home user, movie company, film director, professional oramateur director, etc.). One or more entities within the portion of themovie may be automatically identified as described above. In someinstances, an entity may be manually input by, for example, a sourceprovider or end users. Supplemental content may be received from a firstuser about one or more of the automatically identified entities withinthe first portion of the movie. The supplemental content may include aselection of one or more of the automatically identified entities,and/or additional content provided by a user. A database entry may bestored and/or generated that links the identified entity within theportion of the first movie and the supplemental content about theidentified entity. For example, a selection of one or more automaticallyidentified entities may be received. Supplemental content may bereceived and/or associated with the selected one or more entities. Insome instances, the supplemental content and/or automatically identifiedentities may be associated with a time reference.

In situations in which the systems discussed here collect personalinformation about users, or may make use of personal information, theusers may be provided with an opportunity to control whether programs orfeatures collect user information (e.g., information about a user'ssocial network, social actions or activities, prior media views orpurchases, profession, a user's preferences, or a user's currentlocation), or to control whether and/or how to receive content fromsystems disclosed herein that may be more relevant to the user. Inaddition, certain data may be treated in one or more ways before it isstored or used, so that personally identifiable information is removed.For example, a user's identity may be treated so that no personallyidentifiable information can be determined for the user, or a user'sgeographic location may be generalized where location information isobtained (such as to a city, ZIP code, or state level), so that aparticular location of a user cannot be determined. Thus, the user mayhave control over how information is collected about the user and usedby systems disclosed herein.

The foregoing description, for purpose of explanation, has beendescribed with reference to specific implementations. However, theillustrative discussions above are not intended to be exhaustive or tolimit implementations of the disclosed subject matter to the preciseforms disclosed. Many modifications and variations are possible in viewof the above teachings. The implementations were chosen and described inorder to explain the principles of implementations of the disclosedsubject matter and their practical applications, to thereby enableothers skilled in the art to utilize those implementations as well asvarious implementations with various modifications as may be suited tothe particular use contemplated.

The invention claimed is:
 1. A computer-implemented method comprising:obtaining a portion of a movie; automatically identifying an entitywithin the portion of the movie using a technique selected from thegroup consisting of: face recognition, voice recognition, objectrecognition, individual characteristic recognition, scene breakrecognition, object tracking, speech recognition, and audio recognition;receiving supplemental content from a first user about the identifiedentity within the first movie; subsequent to receipt of the supplementalcontent, receiving an update to the supplemental content about theidentified entity within the movie, wherein the update to thesupplemental content modifies or replaces the supplemental content;receiving a request from a second user for information related to alocation of the second user; and responsive to the request, providing,to the second user, the information related to the location and at leastone item of the supplemental content, from the update to thesupplemental content, related to the location.
 2. The method of claim 1,further comprising storing a database entry that links the identifiedentity within the portion of the movie and the supplemental contentabout the identified entity wherein the database entry is based upon anidentification of the portion of the movie, the supplemental content,and the identified entity to a database.
 3. The method of claim 1,further comprising tracking the entity in the movie throughout aduration that the entity exists within the portion of the movie.
 4. Themethod of claim 1, wherein the entity comprises an audio component ofthe movie.
 5. The method of claim 4, wherein the audio component isselected from the group consisting of: a song, a soundtrack, a voice,and a sound effect.
 6. The method of claim 1, wherein the entitycomprises a visual component.
 7. The method of claim 6, wherein thevisual component is selected from the group consisting of: a scenebreak, a geographic location, a face, a person, an object, a physicalobject, and a text.
 8. The method of claim 1, wherein the supplementalcontent is selected from the group consisting of: a text, an audioentity, a visual entity, a URL, a picture, an advertisement, and alocation.
 9. The method of claim 1, further comprising associating atime reference with the supplemental content.
 10. The method of claim 1,wherein the supplemental content comprises a selection of one or more ofthe automatically identified entities.
 11. A computer-implemented methodcomprising: automatically identifying an entity in a movie using atechnique selected from the group consisting of: face recognition, voicerecognition, object recognition, individual characteristic recognition,scene break recognition, object tracking, speech recognition, and audiorecognition; obtaining supplemental content for the entity identified inthe movie; subsequently, receiving an update to the supplemental contentfor the identified entity in the movie, wherein the update to thesupplemental content modifies or replaces the supplemental content;providing the movie; receiving, from a user, a query for informationrelated to a location of the user, wherein the query is received by aninput method during playback of the movie; retrieving, from a database,the information related to the location and at least one item of thesupplemental content, of the update to the supplemental content aboutthe identified entity, related to the location, wherein the databasecomprises a plurality of automatically identified entities for aplurality of movies and corresponding supplemental content; andproviding, to the user, a response to the query wherein the responseincludes the update to the supplemental content.
 12. The method of claim11 further comprising tracking the entity while the entity is present inthe content.
 13. The method of claim 11, wherein the query is receivedby an input device.
 14. The method of claim 11, wherein receipt of thequery causes playback of the content to pause.
 15. Acomputer-implemented method comprising: receiving supplemental content;storing the supplemental content; obtaining content; automaticallyidentifying an entity in the content using a technique selected from thegroup consisting of: face recognition, voice recognition, objectrecognition, individual characteristic recognition, scene breakrecognition, object tracking, speech recognition, and audio recognition;subsequent to receipt of the supplemental content, receiving an updateto the supplemental content about the identified entity within thecontent, wherein the update to the supplemental content modifies orreplaces the supplemental content; retrieving a portion of the update tothe supplemental content based on the entity; storing a database entrythat links the identified entity with the portion of the update to thesupplemental content while the entity is present in the content;providing, to a user, a portion of the content, the portion of thecontent including the identified entity; receiving, from the user, arequest for information related to a location of the user and based uponthe portion of the content and the identified entity; and responsive tothe request, providing, to the user, the information related to thelocation and at least one item of the supplemental content, of theupdate to the supplemental content, related to the location.
 16. Themethod of claim 15, wherein the supplemental content is selected fromthe group consisting of: a text, an audio entity, a visual entity, aURL, a picture, an advertisement, and a location.
 17. The method ofclaim 15, wherein the content comprises a movie or multimedia data. 18.The method of claim 15, wherein entity comprises an audio component ofthe content.
 19. The method of claim 17, wherein the audio component isselected from the group consisting of: a song, a soundtrack, a voice,and a sound effect.
 20. The method of claim 15, wherein the entitycomprises a visual component.
 21. The method of claim 19, wherein thevisual component is selected from the group consisting of: a scenebreak, a geographic location, a face, a person, an object, a physicalobject, and a text.
 22. The method of claim 15, further comprisingassociating a time reference with the supplemental content.
 23. Themethod of claim 15, further comprising tracking the entity in thecontent throughout a duration that the entity exists within the content.